ChatGPT Responses in Hard Flaccid Syndrome: Exploratory Evaluation

Introduction

Hard flaccid syndrome (HFS) is a rare, acquired disorder that manifests as a semirigid penis when flaccid without stimulation with erectile dysfunction; perineal and penile sensory changes such as burning, coldness, numbness, and pain; urinary symptoms such as painful urination and weak flow; and emotional and psychosocial distress [1]. While the etiology and pathophysiology are still unclear, some studies propose a relation with activation of the pelvic or pudendal hypogastric reflux [2] or past trauma to the associated neurovasculature [3,4].

As evidence is still scarce regarding HFS, treatment options are still limited to conservative management that targets improving psychosocial triggers, pelvic floor muscle relaxation, analgesia, and phosphodiesterase 5 inhibitors for erectile dysfunction [5-7]. These treatment options, while individually beneficial, have not yet been established as an effective standardized guideline for HFS.

As AI becomes increasingly integrated into everyday life, health care has undergone significant transformation, giving patients faster and more efficient access to medical information, diagnostic tools, and treatment options [8,9]. Despite these advancements, significant concerns remain about incorporating AI into health care as errors, limited training data, and hallucinated outputs are still common—particularly in the context of rare conditions [10-13].

Given the rarity of HFS, the limited evidence base, and the exploratory nature of evaluating AI performance in this setting, this study was designed as an exploratory assessment intended to characterize preliminary patterns in the accuracy, readability, and reproducibility of AI-generated information rather than provide definitive performance estimates. As such, this study aims to assess the accuracy, readability, and short-term reproducibility of AI model responses to common questions regarding HFS epidemiology, symptoms and risk factors, diagnostics, and therapy options.

Methods

Input Questions

This exploratory study evaluated the performance of ChatGPT (OpenAI) in addressing patient-generated questions related to HFS. The study questions were developed using authentic patient queries encountered in clinical practice supplemented by questions reported in prior investigations of rare diseases unrelated to HFS [14,15]. This study focused on questions with common use and broad clinical relevance, leaving out any that were nonclinical in nature, insufficiently precise, or tailored to a single patient’s situation.

Responses were generated using the publicly available ChatGPT web interface accessed through a free user account. Sixteen predefined questions were independently submitted as separate prompts in newly initiated chat sessions to minimize contextual carryover between responses. Responses were not regenerated, edited, or refined after their initial generation. The same prompting procedure was repeated 48 hours later to assess short-term reproducibility.

No API access, fine-tuning, or external model modifications were used. Data collection was performed between November 27, 2025 (T1), and December 1, 2025 (T2). No custom instructions were used. Memory settings were not intentionally modified and were left at the account’s default configuration during data collection. Web search functionality was not intentionally invoked during any prompting session. The domains and exact prompts are shown in Table 1. The responses at T1 and T2, copied and pasted directly from the ChatGPT chat, are shown in Multimedia Appendices 1 and 2 respectively.

Response Grading

Every answer was graded independently by 3 English-speaking urologists with expertise in male sexual health. Any discrepancies in grading were further assessed by a blinded fourth urologist. For reporting purposes, a single final consensus accuracy score was assigned to each response after adjudication and was subsequently used for all descriptive analyses, figures, and reproducibility assessments.

Readability of the text responses was assessed using the Gunning Fog Index (GFI), which is a validated scoring system that checks for wordiness and use of complex wording [16].

To assess accuracy, each grader independently analyzed the response thoroughly using a scoring system adapted from the work by Yeo et al [15], classifying responses as correct and comprehensive (4), correct but inadequate (3), having mixed correct and incorrect information (2), or completely incorrect (1). The interrater reliability (IRR) of these ordinal accuracy assessments was subsequently evaluated using the intraclass correlation coefficient (ICC), as described in the Statistical Analysis section. Higher accuracy scores indicated better performance, with grade 4 representing correct and comprehensive responses and grade 1 representing completely incorrect responses. The grades given by each individual reviewer are shown in Multimedia Appendix 3.

Short-term reproducibility was assessed by comparing the final adjudicated accuracy classifications assigned to responses generated at T1 and T2 using identical prompts. Reproducibility was defined as identical final accuracy ratings across both evaluation sessions.

Statistical Analysis

Descriptive statistics were calculated for all outcomes within each domain. Given the small sample size within each domain (n=4 questions), formal normality testing was not performed as such tests have limited interpretability in very small samples. Descriptive statistics are presented as means and SDs. Paired Student independent 2-tailed t tests were used to compare overall readability scores between T1 and T2 across all 16 questions, whereas nonparametric methods were selected a priori for comparisons across domains.

IRR among the 3 independent reviewers was assessed using the ICC. The ICC was selected over κ-based statistics because more than 2 raters were involved and ratings were assigned on a 4-point ordinal scale, for which ICC is commonly used to estimate agreement while accounting for both systematic and random sources of variance. Reliability analyses were performed separately for T1 and T2 using a 2-way random-effects model with absolute agreement assuming that both raters and evaluated items constituted random samples from larger populations.

To examine differences in readability across domains within each evaluation session, the Kruskal-Wallis test was applied. This nonparametric test was chosen due to the small sample size and to avoid assumptions of normality and homogeneity of variance.

The 95% CIs were reported where applicable, and statistical significance was defined as a P value below .05. All analyses were conducted using SPSS Statistics for Windows (version 28.0; IBM Corp). No procedures for handling missing or imbalanced data were required as the analysis was based on complete textual prompts and AI-generated outputs rather than conventional structured datasets. All outputs were generated using the ChatGPT web interface using GPT-4.1, which was available at the time of data collection. No API access, fine-tuning, custom instructions, retrieval augmentation, or model parameter modifications were applied.

Ethical Considerations

This study did not involve human participants, patient recruitment, patient records, identifiable personal information, or any intervention involving living individuals. All analyses were performed exclusively on AI-generated responses to predefined, non–patient-specific prompts submitted by the investigators. Consequently, this work does not constitute human participant research and did not require institutional review board approval or informed consent according to institutional and international research ethics principles governing studies that do not involve human participants or identifiable data. No patient-derived data were collected, stored, or analyzed at any stage of the study.

As this study represents a methodological evaluation of publicly accessible AI outputs rather than an observational or interventional human research study, no formal ethics application was submitted, and no protocol preregistration was required.

Results

Readability of ChatGPT Responses

Across all 16 questions related to HFS, the mean GFI score showed a small numerical decrease from 15.19 (SD 1.68) at T1 to 14.59 (SD 1.66) at T2, indicating largely stable linguistic complexity between the 2 evaluation sessions. Overall readability remained at a level consistent with upper high school to early college comprehension. Mean GFI scores by domain and overall at both evaluation sessions are summarized in Table 2.

When stratified by domain, mean GFI scores at T1 ranged from 14.55 (SD 2.36) to 15.70 (SD 0.91), with the lowest readability complexity observed in the treatment domain. At T2, mean GFI scores ranged from 13.81 (SD 0.85) to 15.23 (SD 2.83), with the greatest variability observed in the symptoms and case presentation domain.

Comparison of Readability Between Evaluation Sessions

Paired analysis comparing readability scores between evaluation sessions demonstrated a small, nonsignificant numerical decrease in GFI from T1 to T2. The mean paired difference (GFIT1 − GFIT2) was 0.60 (95% CI −0.67 to 1.86). This difference did not reach statistical significance (t₁₅=1.01; 2-sided P=.33). The associated effect size was small (Cohen d=0.25, 95% CI −0.25 to 0.75), indicating minimal practical difference in linguistic complexity between responses generated at the 2 time points. These findings suggest that overall readability of ChatGPT responses related to HFS remained stable across evaluation sessions.

Readability Differences Across Domains

Nonparametric comparison of readability across the 4 predefined content domains was performed using the Kruskal-Wallis test. At the first evaluation session (T1), no statistically significant differences in GFI scores were observed between domains (Kruskal-Wallis H₃=1.17; P=.76) despite modest variation in mean ranks across domains. Similarly, at the second evaluation session (T2), GFI scores did not differ significantly by domain (H₃=1.52; P=.68). These findings indicate that the linguistic complexity of ChatGPT-generated responses related to HFS was broadly consistent across the epidemiology, treatment, diagnostics, and symptoms and case presentation domains at both time points.

Accuracy and IRR

The distribution of accuracy grades varied across domains and evaluation sessions, as shown in Table 3. At T1, a total of 43.8% (7/16) of the responses were classified as correct and comprehensive (grade 4), 50% (8/16) were classified as correct but inadequate (grade 3), and 6.3% (1/16) were classified as having mixed correct and incorrect information (grade 2). No responses were classified as completely incorrect (grade 1).

At T2, performance improved further, with 62.5% (10/16) of responses classified as correct and comprehensive and 37.5% (6/16) classified as correct but inadequate. No responses received grades of 1 or 2.

Performance varied according to clinical domain. Epidemiology, treatment, and diagnostics demonstrated predominantly grade 4 classifications at both evaluation sessions, whereas responses related to symptoms and case presentation were more frequently classified as grade 3 (correct but inadequate), suggesting greater difficulty in addressing nuanced symptom interpretation and patient counseling questions. Overall, these findings suggest that ChatGPT performed better when providing general descriptive information than when addressing more complex clinical topics requiring nuanced evidence synthesis and interpretation. The analyses presented below were based on the final adjudicated accuracy score assigned to each response following reviewer consensus.

At T1, IRR demonstrated fair agreement at the level of individual reviewers, with a single-measure ICC of 0.36 (95% CI 0.06‐0.67, P<.01). Agreement improved to a moderate level when ratings were averaged across reviewers, with an average-measure ICC of 0.63 (95% CI 0.17‐0.86, P<.01). At T2, IRR remained statistically significant and showed comparable agreement. The single-measure ICC was 0.33 (95% CI 0.04‐0.64, P=.01), indicating fair agreement between individual reviewers, whereas the average-measure ICC reached 0.59 (95% CI 0.10‐0.84, P=.01), reflecting moderate agreement when reviewer scores were averaged. All ICC estimates were derived using a 2-way random-effects model with absolute agreement. Overall, the IRR of accuracy assessments was fair at the individual reviewer level and moderate when ratings were averaged across reviewers at both evaluation sessions. Internal consistency among reviewers was acceptable at T1 and moderate at T2, as summarized in Table 4. The distribution of accuracy scores for individual questions across domains and evaluation sessions is illustrated in Figure 1.

‎

Reproducibility of Responses

Overall reproducibility of ChatGPT responses between the 2 evaluation sessions was high. Overall accuracy classification stability was high, with 87.5% (14/16) of responses receiving identical final accuracy ratings between T1 and T2. When stratified by domain, reproducibility was 100% in the epidemiology and symptoms and case presentation domains. In the treatment and diagnostics domains, 1 response in each domain demonstrated a change in reproducibility classification between sessions, resulting in a reproducibility rate of 75% for both domains. These findings indicate high short-term consistency in the reproducibility of ChatGPT-generated responses related to HFS across evaluation sessions for most patient-generated questions. Reproducibility outcomes across domains are summarized in Table 5.

Discussion

Principal Findings

This study provides one of the first structured evaluations of a large language model’s (LLM) performance in addressing a rare and poorly characterized andrological condition—HFS. Given the scarcity of high-quality clinical guidance and patients’ increasing reliance on online and AI-generated health information, assessing the accuracy, readability, and reproducibility of ChatGPT responses in this context is both timely and clinically relevant. Overall, ChatGPT demonstrated stable readability and high short-term reproducibility, whereas accuracy varied across clinical domains, supporting its potential as an informational resource but highlighting important limitations in rare diseases.

Performance varied across domains. Epidemiology, treatment, and diagnostics more frequently achieved correct and comprehensive responses, whereas symptoms and case presentation responses were more commonly classified as correct but inadequate, suggesting greater difficulty in addressing nuanced symptom interpretation and patient counseling questions. This pattern aligns with existing meta-analytic evidence on LLM performance: a systematic review and meta-analysis reported moderate accuracy (approximately 56%) of ChatGPT across medical query domains but emphasized significant heterogeneity and lack of standardized evaluation protocols [17]. Other quantitative assessments of LLM medical capabilities show similar constraints: LLMs can perform reasonably in examination-style settings but exhibit hallucinations, incorrect inferences, and poor sourcing outside structured prompts or training data [18]. These limitations are particularly concerning in rare disease contexts where corroborating literature is minimal, and patient decisions rely disproportionately on explanation and context.

Considering the clinical ambiguity surrounding HFS and its rarity, there is marked uncertainty in medical grading of model outputs as what may be labeled accurate by an AI tool could reflect prevailing beliefs, anecdotes, or conjectures rather than evidence. This highlights a central challenge for LLM evaluation in rare conditions where gold-standard clinical guidance does not yet exist.

It is critical to emphasize that model accuracy must be interpreted relative to the evidence base itself. For conditions such as HFS with evolving clinical descriptions and few high-quality studies, a model could correctly summarize what is written in the literature but still provide information with limited clinical utility because the underlying literature is limited.

One strong finding of our study was the high short-term reproducibility observed across repeated prompts. This stability is consistent with research showing that modern LLM outputs can be remarkably consistent in controlled settings when the prompt and model version are held constant [19]. However, reproducibility alone does not equate to factual correctness. A model that consistently reproduces incorrect reasoning or unvalidated mechanisms provides false reassurance rather than reliable guidance. This distinction reinforces that stability in outputs is a necessary but insufficient quality for medical use. Importantly, reproducibility and accuracy classification stability were evaluated as separate outcomes. Reproducibility in this study reflected the stability of expert-assigned accuracy classifications across repeated prompts rather than an assessment of semantic similarity between paired responses.

LLM outputs tended to be linguistically coherent and structured, which may increase perceived credibility among lay users. Prior work on health information retrieval has shown that the readability and perceived coherence of a source, whether AI or human, can influence perceived trustworthiness and intention to act even when the content is inaccurate [20]. Such effects raise ethical concerns when disseminating medical advice in high-anxiety contexts such as sexual dysfunction.

Beyond the findings of the present study, an important concern consistently highlighted in the literature is AI hallucinations, wherein generative models produce plausible but erroneous or fabricated content. Surveys of hallucination research highlight that even advanced models continue to generate inaccurate details and invented assertions absent direct evidence [21]. Hallucinations are strongly observed when models encounter low-data or ambiguous domains, as is the case with HFS. In addition, empirical work outside of health care contexts shows that LLMs often prioritize fluent agreement over factual accuracy when presented with illogical or poorly defined questions [22]. These intrinsic limitations underscore the caution that medical professionals and educators have expressed in deploying chatbot responses directly as medical advice, particularly when sources cannot be traced back to validated clinical evidence.

A qualitative review of several responses further illustrates these limitations. For example, when asked about the global prevalence of HFS, ChatGPT occasionally provided speculative statements regarding the number of affected individuals despite the absence of robust epidemiological data in the literature. Similarly, when discussing available treatment options, some responses implied a greater degree of therapeutic consensus than currently exists despite the absence of standardized management guidelines. Although these statements were often accompanied by appropriate caveats regarding limited evidence, they may nevertheless convey an unwarranted sense of certainty to patients seeking information about a poorly characterized condition. These examples underscore how LLMs may generate plausible and coherent narratives that extend beyond the available evidence base.

Several limitations should be acknowledged when interpreting these findings. First, the evaluation was limited to a single LLM and a single model version at a specific point in time; as such, the results may not be generalizable to other AI systems or to future iterations that may incorporate improved medical grounding or retrieval mechanisms. Second, although expert review was used as the reference standard for accuracy, the absence of consensus definitions, validated diagnostic criteria, and high-quality therapeutic evidence on HFS inherently limits the objectivity of accuracy assessments and likely contributed to only fair to moderate IRR. Third, the limited number of questions (16 in total; 4 per domain) reflects the exploratory nature of this study and may not fully capture the breadth, complexity, or heterogeneity of patient concerns related to HFS. Consequently, the findings should be interpreted as preliminary and hypothesis generating rather than definitive estimates of AI performance. Additionally, certain patient-generated questions may overlap across clinical domains, and categorization was necessarily pragmatic rather than absolute. Finally, this study did not evaluate how patients interpret, emotionally respond to, or act upon AI-generated information, an important limitation given the psychosocial vulnerability often associated with sexual dysfunction and chronic pelvic pain syndromes. These factors underscore the need for cautious interpretation of AI performance metrics in rare disease contexts.

Future research should build on these findings by expanding evaluations to include multiple LLMs and repeated assessments over longer time intervals to better characterize temporal variability and model evolution. Incorporating retrieval-augmented architectures grounded in peer-reviewed medical databases may help reduce hallucinations and improve transparency of evidence sourcing, particularly for rare conditions with limited primary data. Further studies should also examine patient-centered outcomes, including comprehension, perceived credibility, anxiety modulation, and downstream health behaviors following exposure to AI-generated information. In addition, the development of standardized evaluation frameworks tailored to rare diseases accounting explicitly for evidence scarcity and clinical uncertainty would improve comparability across studies and reduce subjectivity in expert grading. Ultimately, interdisciplinary collaboration between clinicians, AI developers, and ethicists will be essential to define appropriate use cases for AI-driven health information tools and ensure that such technologies augment rather than undermine evidence-based patient care.

Conclusions

In this exploratory evaluation, ChatGPT demonstrated stable readability and high short-term reproducibility when addressing patient-generated questions regarding HFS, although response accuracy varied according to clinical domain and occasionally extended beyond the available evidence base. These findings highlight both the promise and the current limitations of LLMs as accessible sources of health information for rare diseases. As AI becomes increasingly integrated into patient information seeking, robust validation frameworks, expert oversight, and continued refinement of medically grounded models will be essential to ensure that these technologies complement rather than replace evidence-based clinical care.